Skip to content

status: fix FSMonitor history and clean-proof gaps - #74

Open
ttaylorr-oai wants to merge 29 commits into
codex-unstablefrom
tb/codex/fsmonitor-hardlink-inodes-unstable
Open

status: fix FSMonitor history and clean-proof gaps#74
ttaylorr-oai wants to merge 29 commits into
codex-unstablefrom
tb/codex/fsmonitor-hardlink-inodes-unstable

Conversation

@ttaylorr-oai

@ttaylorr-oai ttaylorr-oai commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Current candidate

Head: 5f1339f46d. This pull request remains review-only; do not merge or
enable auto-merge
. Fresh approval is required for this head before
controller admission.

Two additional patches address command patterns that prevented proof reuse:

  • Recognize empty filter.<driver>.clean and process with
    required=false as complete read-side disabling, without requiring a
    smudge override. Scoped FSMonitor reuse still requires a filter-free scope.
  • Certify a newly created worktree index under its mandatory writer lock,
    including when GIT_OPTIONAL_LOCKS=0. Later read-only status calls reuse
    the paired proof without writing the index or scanning tracked files.

Regression controls cover active filters, provider errors and resets,
post-checkout edits, partial overrides, and both hash algorithms.

Focused native validation passed:

  • t7519-status-fsmonitor.sh: 123 tests
  • t7527-builtin-fsmonitor.sh: 200 tests
  • t7530-status-clean-sidecar.sh: 67 tests
  • worktree add/prune/list/move/config and reset suites
  • unit tests, including the 406-case Clar suite; test lint and diff checks

Hosted push and pull-request CI each completed with 45 successful jobs and
two expected skips. Style and whitespace checks passed.

A fresh clean developer-mode full suite passed: 33,507 successes, zero
unexpected failures, and 378 known breakages. Full status and FSMonitor
suites also passed at both commit boundaries, as did a separate incremental
build through both patches.

Two executable Git workflows passed on each of macOS and Linux, covering
staging, stash, conflicting rebase, linked worktrees, merge, and worktree
removal. Optimized results matched conservative oracles, and the required
read-only checks left the index unchanged. The macOS workflows used the
same sealed artifact as the latency campaign.

Qualification remains open. An earlier incremental full-suite run failed
the linked-worktree stash-history test in t7519. Its failing assertion
was not captured. Subsequent clean, incremental, and paired stress runs
pass; branch traces do not support the proposed cause in these patches.
Three further full t7519 repeats per commit also passed. The simulated
review identified no supported source fix; the original failure remains
recorded and unexplained.

Same-base macOS latency qualification is still open. The corrected
four-repository run passed its oracle, provider-fence, identity, and
read-only-index checks, but did not meet the mean-latency thresholds for
clean writable and dirty read-only status. Median and p95 limits passed.
No samples were discarded and no threshold was relaxed.

A bounded diagnostic captured multi-second stalls in both builds before
Git initialized its process clock; the exact startup cause is unresolved.
These observations are not a latency pass. No release will be admitted
before the remaining latency qualification and fresh approval.

Earlier implementation and validation notes

Summary

  • reject missing clean-status snapshot paths before calling open(2)
  • serialize asynchronous simple-IPC start and stop transitions, make the
    shutdown wake reliable across EINTR, and contain SIGPIPE during gentle
    client writes
  • flush the Darwin FSEvents stream and drain its serial callback queue before
    accepting a query boundary, with bounded timeout, daemon retirement, token
    generation checks, and request coalescing

Controller scope

This review-only pull request presents one prerequisite and two topic patches
on top of 1293167e46. Approve it for controller admission, but do not merge
it into codex-unstable.

Performance

Overlapping Darwin queries whose cookies are already registered share one
provider fence. A timeout or shutdown intersection falls back conservatively
and retires the daemon. Paired same-base latency qualification is being run on
this exact source and binary; this description does not make a latency claim
before that evidence completes.

Validation

At commit 7bed8a334f:

  • clean build
  • t0052-simple-ipc.sh (11/11)
  • immediate-close client stress (200/200 on one socket path and 200/200 on
    unique paths)
  • forced-EINTR shutdown wake stress (50/50, with 50 verified injections)
  • test and diff checks

The preceding candidate, 08f907411e, completed unit tests (406/406),
t7519-status-fsmonitor.sh (110/110),
t7527-builtin-fsmonitor.sh (194/194),
t7530-status-clean-sidecar.sh (59/59), and the
ASan+UBSan clean_status_manifest unit tests (4/4). The only tree change from
that candidate is replacing an open-coded allocation in
t/helper/test-simple-ipc.c with CALLOC_ARRAY() as required by static
analysis. Production sources, FSMonitor tests, and the FSMonitor topic
patch-id are unchanged.

Exact-head hosted CI, same-base latency qualification, and executable workflow
qualification are in progress.

@ttaylorr-oai
ttaylorr-oai marked this pull request as draft August 26, 2026 18:53
@ttaylorr-oai
ttaylorr-oai removed the request for review from dreynaud-oai August 26, 2026 18:53
@ttaylorr-oai
ttaylorr-oai marked this pull request as ready for review August 26, 2026 19:17
@ttaylorr-oai
ttaylorr-oai force-pushed the tb/codex/fsmonitor-hardlink-inodes-unstable branch from f5f8eaf to 0a572e5 Compare August 27, 2026 04:33
with_lock__wait_for_cookie() gives a filesystem provider one second to
report a synchronization cookie. A healthy FSEvents stream can miss
that deadline while macOS is under load. The daemon then returns a
trivial response, and status scans the entire index even though event
delivery is still making progress.

4b1c56a (fsmonitor: flush pending FSEvents before cookie wait,
2026-07-21) requested an asynchronous flush on every Darwin query but
kept the same one-second deadline. f439708 (Revert "fsmonitor: flush
pending FSEvents before cookie wait", 2026-08-17) reverted it after a
matched 48-query test still saw 12 timeouts in each arm. Avoid restoring
that unqualified hot-path request.

When the initial Darwin wait expires, request an asynchronous FSEvents
flush and wait one more bounded interval. Successful queries retain the
original wait and do not issue a flush or extend their deadline. The
asynchronous call cannot block on the callback while the client holds
main_lock. If the provider stays silent, retain the existing
trivial-response fallback after the retry.

Add a test-only callback delay to exercise both outcomes: a 1.2-second
delay is recovered, while a 2.5-second delay still reaches the bounded
fallback.
The daemon currently assumes that each client which advances an
FSMonitor token also updates the repository's canonical index. That
does not hold for commands using GIT_INDEX_FILE. A private index can
advance the daemon past the canonical index's token and cause the
canonical index's next query to receive a global invalidation.

Keep a deduplicated overflow batch instead of discarding old paths.
Clients at the overflow sequence still get an exact delta. Older
clients get a conservative union of paths, which may overreport but
cannot miss a change.

All paths are interned. Keep a pointer-identity hash set with the
overflow batch so later compactions hash only newly retired paths,
rather than rebuilding a set over the daemon's lifetime history.

Add a regression which advances a private index repeatedly, verifies
that compaction remains deduplicated, and then checks that a read-only
canonical status reports both changed files without a trivial response.
Retired batches are collapsed into a path-only overflow set.  That keeps
old indexes complete, but it loses the sequence in which each path was
last observed.  A client that consumed an inode event can therefore see
it again after another index compacts the batch list, causing repeated
hard-link scans.  Unpinned batches have a zero pinned time and are also
eligible for compaction immediately despite the default grace period.

Do not use unpinned batches as truncation boundaries.  Record the newest
original batch sequence for every overflow path, and filter overflow
responses against the client's requested sequence.  The normal batch
walk remains unchanged; sequence lookups are confined to overflow
responses.

Cover both the default retention grace and the cross-index hard-link
case.  The latter persists a nonzero checkpoint, compacts through a
private index, and verifies repeated canonical reads do not rescan or
fall back to global invalidation.
The delayed-cookie tests send the v1 timestamp token "0" and only check
that the response is nonempty.  Both recovery and fallback can satisfy
that assertion with the same trivial response, so the tests do not
distinguish a rescued cookie from a token-generation reset.

Send a deterministic valid v2 token instead.  Verify that the 1200ms
case preserves its token generation without a global invalidation, while
the 2500ms case changes generation and sends the fallback invalidation.
215845a (fsmonitor: preserve authenticated proofs across ordinary
commands, 2026-08-15) enabled the clean-status history handoff for
merges, but excluded invocations where fast_forward was FF_NO.
Requested merge topology does not determine whether the resulting index
is semantically safe.  A clean non-fast-forward merge can carry the same
authenticated FSUC/FSCF state as a fast-forward merge.

As a result, --no-ff, --no-ff --no-commit, and merge.ff=false all
dropped FSUC and reduced the FSCF flags from 15 to 9 after a clean
merge.  Each subsequent read-only status invalidated the external
history and rescanned the semantic manifest.

Enable the handoff for every merge using the canonical index.  Conflict
handling still invalidates unsafe proofs, and explicit alternate indexes
remain excluded.  Cover all three non-fast-forward forms, repeated
read-only status calls, conflicts, and alternate indexes.
An exact clean status can repair a stale FSMonitor checkpoint or cached
stat data while it scans.  The repair requires an index write, so the
existing issue path leaves no clean sidecar behind.  Read-only callers
then repeat the full scan until a second writable exact status publishes
the proof.

After the repair is written and resumable history is durable, install a
sidecar bound to the rewritten index.  Keep optional-lock-disabled
commands read-only, preserve the literal exact-command restriction, and
do not extend sidecar support to linked worktrees.

Cover repeated read-only scans after a legacy daemon replacement, the
single writable index repair in main and linked worktrees, and the next
read-only sidecar hit in the main worktree.  Keep option-bearing status
commands ineligible for proof publication.
A configured pull can discard each layer of authenticated status history
even when worktree inputs remain unchanged. Command-scoped protocol and
HTTP settings change the config digest, directory events with more than
64 tracked descendants reject the semantic proof, and a fast-forward
which adds an indexed directory drops the paired untracked cache. The
next status can consequently preload and refresh the full index.

Treat command-scoped protocol and HTTP settings as transport-only. For a
large directory event, authenticate each distinct attribute source once
instead of rejecting the cone outright. When a checkout adds tracked
paths, retain the paired untracked cache and replay those additions
through its existing invalidation path.

Cover configured pulls in main and linked worktrees, large directory
events, nested attribute-source changes, and branch switches which add
tracked directories. The conservative full-scan fallback remains in
place when an attribute source changes.
Configured pulls preserve FSMonitor clean proofs when checkout can
authenticate every index change.  Tracked policy files were an
exception: adding or replacing .gitattributes or .gitignore made the
generic semantic transfer reject the whole proof.  Later read-only
status commands then had to rescan the worktree and could not restore
the paired untracked proof.

Let checkout retain history across regular policy-file changes that it
writes itself.  Attribute changes refresh the worktree manifest before
the provider boundary is rebound, and fail closed if that refresh cannot
authenticate the new sources.  Keep the existing untracked-cache
invalidation for ignore changes, and transfer that cache only while the
full tracked proof remains current.

Exercise configured fast-forward pulls in main and linked worktrees.  A
required-filter control also verifies that changed attributes invalidate
the affected tracked entry instead of certifying it.
A clean status proof can survive a pull only when its configuration,
tracked-file state, FSMonitor token, and paired untracked cache still
describe the resulting worktree. Command-scoped push transport settings
were included in the configuration fingerprint. Checkout could also
discard the untracked proof for policy-file changes or leave events from
its own worktree writes outside the proof.

The next diff, write-tree, or status then repeated tracked and untracked
work. With optional locks disabled, status could not publish the repair,
so each invocation paid the same cost.

Treat push.negotiate and remote.*.pushurl like other command-scoped
transport settings. Preserve the paired untracked cache across checkout,
invalidate only affected policy scopes, and authenticate distinct
attribute-source directories before transferring semantic history.

For checkout, reset, merge, and sequencer worktree updates, write a
provisional index under the existing lock, consume the daemon events
caused by the update, and certify the result against that locked index
before the final write. This also covers stash cleanup through its hard
reset. Alternate indexes, split or sparse indexes, unsafe filter or
manifest state, and incomplete stat data still fall back.

Cover configured pulls, root and nested policy changes, main and linked
worktrees, rebase, reset, stash, checkout, and repeated read-only
status.
The Linux listener queued every inotify event as a file pathname.
Directory events therefore lacked the trailing slash used by semantic
invalidation. After an owned worktree update retained a clean proof, a
following read-only status could not close those events against it. The
command scanned all tracked entries. With core.preloadIndexBulk enabled,
this work appears as statx calls instead of lstat counters.

Format Linux worktree events through
fsmonitor_format_worktree_paths() and use IN_ISDIR to preserve their
directory identity. Advertise directory metadata support and mark Linux
tokens so clients replace daemons using the old event format.

Concurrent clients can see an expected connection reset while one client
replaces a stale daemon. Silence that diagnostic only for gentle IPC
reads, which already reconnect, without changing ordinary IPC error
handling.

Cover both bulk preload modes plus single and concurrent daemon
replacement.
Bulk index preload can defer content and conversion checks to the diff
that normally follows refresh_index(). repair_fsmonitor_proof() only
refreshes the index before deciding whether to persist a clean proof; it
does not run that diff. With core.preloadIndexBulk enabled, a pull or
rebase that changes .gitattributes or .gitignore can therefore leave
tracked entries dirty after the writer reports a successful repair.
Repeated read-only status calls cannot persist the missing repairs.

Do not request deferred bulk results in the writer-repair path. This
keeps the ordinary status and diff bulk path unchanged while forcing the
exceptional repair to finish its tracked checks before certifying and
writing the proof.

Enable bulk preload in the existing fast-forward, policy-file, and
sequencer writer tests. They verify targeted refreshes and two
subsequent read-only status calls without scans or index writes.
A writable Git command can leave a complete FSMonitor proof in a
repairable state when it changes policy files, adds or removes an
intent-to-add entry, or delegates the final index write to a child
process. The existing repair path assumed that the manifest and
untracked cache remained closed. The sequencer also kept its stale
in-memory index after git commit rewrote the canonical index. Stash
operations and completed rebases could therefore drop FSUC or overwrite
the child's newer token. Read-only status could not persist the repair
and repeated tracked or directory work.

Let index-only writers refresh changed manifests and rebuild the paired
untracked cache against the provisional locked index. Preserve unrelated
history for safe intent-to-add changes and unmerged non-attribute paths,
then reload the canonical index after child writers before repairing it.
Active filters and unresolved structural indexes still fall back.

Linux can report an event for a watched directory without a child name.
Keep the watched directory in that case, encode its token capabilities
in the order understood by Linux clients, and serialize incompatible
daemon replacement on Linux as on macOS.

Cover stash creation and application, policy-file updates, ordinary and
--rebase-merges conflict completion, cherry-pick's deliberately weaker
tracked-only proof, nameless inotify events, and primary and linked
worktrees. Repeated optional-lock-free status calls must not rewrite the
index or rescan tracked entries.
@ttaylorr-oai
ttaylorr-oai force-pushed the tb/codex/fsmonitor-hardlink-inodes-unstable branch from 0710046 to 1ecb949 Compare August 27, 2026 21:04
snapshot_open() initializes its output before passing the path to
open_nofollow(). The manifest builder can ask it to pin a synthetic
repository while rejecting a sparse index. Such a repository need not
have an index path.

Ordinary Linux happened to return EFAULT from open(NULL), but passing
NULL violates the contract of open(2) and aborts under UBSan.

Reject NULL and empty paths after initializing the snapshot. The
existing clean-status-manifest sparse-index test exercises the
fail-closed result.
@ttaylorr-oai
ttaylorr-oai force-pushed the tb/codex/fsmonitor-hardlink-inodes-unstable branch from 1ecb949 to 08f9074 Compare August 27, 2026 22:24
766fce6 (simple-ipc: split async server initialization and
running, 2024-10-08) separated server initialization from startup so
owners could finish setup before accepting clients. But start and stop
still inspect lifecycle flags without shared synchronization. A late
start can therefore release workers after cleanup requested shutdown,
and concurrent cleanup paths can repeat the stop sequence.

Two transport races compound that problem during daemon replacement.
An interrupted shutdown wake can publish the shutdown transition
without waking accept(), leaving the stop path hung. A client whose
accepted socket is closed before its request write can die from
SIGPIPE before the gentle EPIPE recovery runs.

Serialize the first start and stop, queue the complete shutdown wake
before publishing that transition, and contain SIGPIPE within gentle
client writes while preserving the caller's signal state. Exercise
late startup, repeated stop, interrupted wakeups, and concurrent writes
to a peer that closes immediately after accept().
The Darwin daemon treats delivery of its cookie-file event as proof
that all earlier worktree changes have been published. That assumes the
callback containing the cookie cannot overtake logically older work.

A retained FSEvents trace disproves that assumption. The cookie
callback completed before a later callback published removals that had
happened before the cookie was created. A status query could therefore
answer from incomplete event history and report a dirty worktree as
clean.

After the ordinary cookie wait, ask a long-lived worker to flush the
FSEvents stream and then drain its serial callback queue. The flush
schedules provider events; the queue drain waits for those callbacks
to finish publishing. Accept the boundary only when the cookie was
seen in the same token generation, and coalesce overlapping requests
onto a single fence.

If the bounded fence times out or intersects shutdown, return a
conservative result and retire the daemon before unsafe stream
teardown. Advertise the stronger boundary as a capability and token
suffix so new clients replace unfenced daemons while older clients
retain prefix compatibility.

Exercise split and blocked callbacks, timeout replacement, generation
reset, listener shutdown, concurrent coalescing, second-wave requests,
rename and cache scopes, and protocol compatibility. Keep status proof
tests outside the split-index matrix where that proof is deliberately
disabled, and materialize externally restored tokens before raw-index
helpers consume them. The provider fence adds work to each Darwin
query, while overlapping queries share a fence when their cookies are
already registered.
@ttaylorr-oai
ttaylorr-oai force-pushed the tb/codex/fsmonitor-hardlink-inodes-unstable branch from 08f9074 to 7bed8a3 Compare August 27, 2026 22:40
ae161e8 (fsmonitor: validate builtin daemon responses before
applying them, 2026-07-10) validates each worktree path with
verify_path(). That helper enforces index-entry rules and rejects a
.git component anywhere in a path. Filesystem providers can
legitimately report such a component for an untracked nested
repository.

The client therefore rejects the entire response after an event such
as scratch/.git/file, forcing a full worktree scan. Commands that need
a current provider boundary cannot persist a clean proof from that
query.

Validate the narrower daemon-response contract instead: require a
relative path with nonempty, non-dot components and at most one
trailing separator. Keep rejecting absolute and traversal paths, but
allow .git components that already exist in the worktree.

Cover the parser directly and exercise a real Linux daemon event
against an optional-lock-free status oracle.
@ttaylorr-oai
ttaylorr-oai force-pushed the tb/codex/fsmonitor-hardlink-inodes-unstable branch 2 times, most recently from ad39dc8 to 2ebd9f8 Compare August 28, 2026 05:07
fbd7a23 (rebase: introduce and use pseudo-ref REBASE_HEAD,
2018-02-11) records the commit currently being replayed. The
sequencer normally deletes that ref before executing each todo item.

When the final item stops for a conflict, rebase --continue commits
the resolved result. pick_commits() then reaches the end of the list
without entering another iteration and removes the rebase state
directly. The merge backend skips finish_rebase() because the
sequencer owns cleanup, so REBASE_HEAD survives a successful rebase.

The same path also strands the fsmonitor proof when index.skipHash is
enabled. After committing the resolution, the sequencer reloads the
canonical index and repairs its proof through a close-only index.lock
witness. A skip-hash witness has a null trailer and a fresh file
identity, so proof-epoch validation cannot bind it to the in-memory
index. Rebase succeeds without FSUC, and read-only status cannot repair
it.

Delete REBASE_HEAD whenever interactive-rebase state is removed. This
matches finish_rebase() cleanup and also avoids retaining a ref for an
explicitly quit operation. Propagate a failed deletion so rebase does
not report success after leaving the stale ref behind.

Give only PROVISIONAL_LOCK witnesses a real checksum. The final index
rewrite continues to honor index.skipHash, preserving the normal index
write fast path while giving proof repair an authenticated epoch.

Extend the final-conflict test to require REBASE_HEAD during
resolution, its removal after completion, and a reported failure when
the ref cannot be deleted. Exercise skip-hash proof repair after a
clean-prefix, conflicted replay in primary and linked worktrees, with
plain and configured-filter repositories.
@ttaylorr-oai
ttaylorr-oai force-pushed the tb/codex/fsmonitor-hardlink-inodes-unstable branch from 2ebd9f8 to 16d98d8 Compare August 28, 2026 05:34
A proof repair can close one provider token, collect untracked results,
reopen the token, and close it again with the same struct wt_status. If
the first closure published untracked output, the second closure tries
to publish another snapshot over it and hits:

    BUG: publishing untracked results over collected status

This is reachable from stash pop when an index writer repairs a complete
FSMonitor proof while an untracked path is present.

Before closing a required new token, discard output explicitly marked as
coming from an earlier authenticated token closure or bulk preload.
Keep the BUG for ordinary caller-collected results, which must not be
silently overwritten. Extend refresh invalidation to discard both
authenticated forms as well.

Allow the scripted provider to opt into proof repair, and add a
regression covering the two-token stash path with visible untracked
output.
An fsmonitor provider can reset while merge is reading an index with an
authenticated clean-status proof. The reset leaves that proof available
for revalidation, but merge updates the worktree before repairing it.
The resulting index can lose FSUC after a clean merge. Read-only status
cannot persist the missing proof, so every later status falls back.

A multi-strategy merge can lose the same history after preparation. An
external strategy may replace the index before it declines or reports a
conflict. restore_state() then reloads that index while rewinding the
worktree. A later built-in strategy sees the original repair decision,
but no longer has the paired proof from which to repair.

Resolved conflicts expose a separate instance of the same failure.
merge clears the resolve-undo extension before updating the worktree.
That removal sets RESOLVE_UNDO_CHANGED, which prevented checkout from
transferring an otherwise current proof. The result had neither a live
provider token nor a pending token from which the writer could repair.

Revalidate an authenticated proof before a non-fast-forward merge
updates the worktree. Repair it before built-in results are published,
after successful external strategies, and after each restore_state()
rewind. Permit transfer after the resolve-undo map has been cleared,
since removing that optional extension changes neither tracked entries
nor worktree contents. Continue rejecting a live resolve-undo map.

A repaired writer stats only entries that lack provider validation or
stat data before certifying the new index. Fast-forward merges retain
their existing path, while conflicts continue to fail closed.

Cover built-in ort with and without retained resolve-undo history,
trivial and content-level resolve merges, an external strategy that
declines, and one that leaves a three-stage conflict before a clean ort
retry. Verify that clean results remove resolve-undo data, keep a paired
proof, and leave repeated read-only status unable to rewrite the index.
378744b (status: reuse closed proofs for scoped queries,
2026-08-11) taught the untracked cache to reconcile a provider-reported
direct child without reopening its directory.  A valid cached directory
can still have a null exclude_oid when its existing contents are all
tracked, since traversal never needed to load its tracked .gitignore.
prep_exclude() interprets that null OID as proof that no per-directory
exclude exists.  The targeted refresh can therefore report a newly
created ignored file as untracked.

Before refreshing a provider-dirty cached directory with a null exclude
OID, use its tracked exclude as the expected identity and load the
worktree source.  Prefer the exact stage-zero entry, then look for a
case-folded alias on case-insensitive worktrees.  add_patterns() still
opens and hashes the actual source when only an alias exists, so a case
collision can only force invalidation.  Use the empty-blob ID for an
unmerged, non-regular, removed, or intent-to-add match so that it forces
a source read and conservative invalidation unless the source is truly
empty.

Cover both exact and case-folded tracked excludes with read-only status
calls.  They must match a cold oracle without opening the directory or
writing the index, and a changed source must invalidate the cache.  Also
pin the sparse-index boundaries: an in-cone event retains targeted
refresh without expansion, while a vivified outside-cone source takes
the existing conservative expansion path.
@ttaylorr-oai
ttaylorr-oai force-pushed the tb/codex/fsmonitor-hardlink-inodes-unstable branch from fd910c1 to b4d251b Compare August 30, 2026 00:24
The provider fence added in 7bed8a3 (fsmonitor: fence Darwin
callbacks before answering queries, 2026-08-27) calls
FSEventStreamFlushSync() from a long-lived worker. The client gives
that worker one second before it retires the daemon.

Under sustained status traffic, the synchronous provider call can
cross that deadline and return immediately afterward. The timeout
still forces a daemon restart, and the next status conservatively
scans the worktree. A retained trace showed this turning a clean
status into a 17-second outlier.

On local APFS and HFS volumes, register sticky vnode watches on each
watched root and its canonical ancestors before starting the FSEvents
stream. Use FSEventStreamFlushAsync() and wait until the callback has
published through its returned event ID and a serial queue barrier.
The worker waits on its existing condition variable, so the bounded
timeout can interrupt it without racing an uncancellable provider
call.

WatchRoot notifications have event ID zero and cannot be represented
by that monotonic token. Treat the kqueue poll as the fence's
linearization point, and reject the fence if any watched namespace
edge occurred or a watched root changed identity. This also covers a
root or ancestor moving away and back before the fence completes.

Fall back to the synchronous provider fence when the vnode proof
cannot be installed, preserving the existing conservative behavior on
unsupported filesystems and resource failures.

Exercise the positive event-ID wait, the zero-ID rename ABA, the
synchronous fallback, and 512 consecutive read-only status calls.
Require every stress-test request to complete without restarting the
daemon.
@ttaylorr-oai
ttaylorr-oai force-pushed the tb/codex/fsmonitor-hardlink-inodes-unstable branch from b4d251b to 837f89d Compare August 30, 2026 00:48
An authenticated clean-status sidecar is bound to the identity of the
index file it certifies. A stash push can restore a complete FSMonitor
and untracked-cache proof after its child processes rewrite the index,
but the existing sidecar still names the old inode. The next read-only
status rejects it and takes the slower history path even though stash
left the worktree clean.

Remember whether stash started with a regular, singly linked sidecar.
When optional locks are available and no post-index-change hook is
configured, retain the status data gathered by proof repair, commit and
reread the final index, then issue a replacement sidecar from that same
certifying scan. This avoids a second worktree traversal while binding
the proof to the final index identity. Other stash paths keep the
existing repair behavior.

Cover a scoped push with a subsequent read-only status that must take
the clean-proof fast path. Let the existing writer-proof test accept
both authenticated sidecar hits and coherent-history reuse, since both
are valid read-only fast paths.
An authenticated clean-status sidecar is bound to the identity of the
index file it certifies. During an interactive rebase, the child commit
run by "rebase --continue" can replace that index. The worktree and
index are clean when the replay finishes, but the remaining sidecar
still names the old inode. The next read-only status rejects it with a
fast-index-mismatch and falls back to the slower history path.

Remember whether the rebase started with a regular, singly linked
sidecar and persistent FSMonitor proof history. After the replay
finishes successfully, reread the final index and use the existing
writer-proof repair to publish a replacement sidecar. Only do so when
optional locks are available and no post-index-change hook is
configured, matching the existing stash guardrails.

Move the sidecar-presence check and sidecar-capable repair helper into
wt-status so stash and sequencer can share them. Cover a conflicted
interactive rebase whose continuation must publish a replacement
sidecar and whose next read-only status must hit it.
The clean-status sidecar path accepts only the main worktree. A newly
created linked worktree therefore cannot publish a proof after a full
clean scan. Later read-only status commands rescan the worktree even
though each linked worktree has its own index and sidecar path.

Accept a linked worktree only when its per-worktree gitdir remains
registered in the common directory and the registered path names the
current worktree. Reject an alternate worktree paired with a linked-
worktree gitdir; the repository fingerprint continues to bind the
resolved worktree, gitdir, common directory, index, and filesystem
identities.

New worktrees are commonly probed with "git status --short". Let the
exact top-level --short and -s forms certify empty output. A fresh
worktree index can still be racy, so write and re-read it before saving
resumable history and issuing the sidecar. This binds both proofs to the
new on-disk index epoch.

Cover issuance and optional-lock-free reuse in a registered linked
worktree, and verify that an impostor worktree sharing its gitdir falls
back.
Stash and rebase may repair and reissue a clean-status sidecar after
their primary operation has completed.  A lock, read, or proof-repair
failure in that optional work currently replaces the successful command
result.  The command then reports failure even though it has already
updated the repository and worktree.

The repair path also commits its updated index before rereading it and
issuing the sidecar.  Another writer can replace the index in that gap.
The old clean scan could then be bound to the replacement index.
A later status could hide a newly staged change.

Treat sidecar repair as best-effort after stash and rebase complete.
Retain a descriptor-backed snapshot of the index produced by each
postwrite clean scan.  Require it to match both the reread index state
and the canonical index path before installing the sidecar.  A failed
repair or intervening write therefore omits the cache and falls back to
ordinary status without changing the primary command result.  The normal
sidecar-hit path remains unchanged.

Cover lock contention after successful stash and rebase operations, and
replace the index at deterministic postwrite barriers in both repair and
status issuance paths.
@ttaylorr-oai
ttaylorr-oai force-pushed the tb/codex/fsmonitor-hardlink-inodes-unstable branch from 23f8d90 to d5c2374 Compare August 30, 2026 11:11
Clean-status sidecars require a durable index identity on local APFS.
The history behavior is still valid on other filesystems, but two tests
required index.csts after their substantive assertions passed and
failed during cleanup on Linux.

Hardlink metadata events can also arrive before status refreshes the
index. If that refresh creates a racy index, status may conservatively
withhold a sidecar until a later scan restores its process-local proof.
Requiring immediate reissuance made the test depend on provider timing.

Gate sidecar removal on local APFS. For the racy-index case, accept
either immediate reissuance or the conservative fallback, but require
clean output and recovery to a new proof within three status calls.
@ttaylorr-oai
ttaylorr-oai force-pushed the tb/codex/fsmonitor-hardlink-inodes-unstable branch 5 times, most recently from a43fd03 to 02e1d29 Compare August 31, 2026 07:30
b65fc91 (status: retain the identity of an index it rewrites,
2026-08-17) lets status keep a race-proof receipt for an index it
rewrites. Receipt preparation duplicates the writer descriptor so it can
hash the final bytes, and requires the in-memory checksum to match the
configured null trailer.

Three owned write paths can leave an otherwise valid index without a
usable clean proof. A worktree-update repair first writes a checksummed
provisional index, then reopens the lockfile write-only before the final
skipHash write while the index still records the provisional checksum.
Receipt preparation rejects both states, so scoped stash cannot publish
a sidecar for the index it installs.

Worktree add creates its linked index before the new worktree has a
closed FSMonitor provider epoch. The index has FSMonitor and
untracked-cache extensions, but lacks the authenticated clean-config
proof. Later read-only status processes remain correct, but cannot
persist that proof and repeat the full fallback on every invocation.

A post-checkout hook can also change worktree-specific configuration.
Checking the invoking worktree's settings after the hook can therefore
skip priming when the hook enables FSMonitor and the untracked cache
only in the new worktree.

A clean non-fast-forward merge repairs its authenticated index proof
before committing, but leaves the existing sidecar bound to the old
index and HEAD tree. The next read-only status rejects it with a
fast-index-mismatch and scans the semantic manifest. Only a later
writable status can replace the stale sidecar.

On Apple, add a read-write reopen operation only for the provisional
index lock so receipt preparation can read the final index. Fall back to
the original write-only reopen when read access is unavailable, allowing
the write to succeed without a receipt. Other platforms retain the
write-only reopen. Finish provisional writes through a cold helper that
restores the null object ID before the receipt-aware write.

After worktree add successfully runs the post-checkout hook, read the
linked worktree's effective FSMonitor and untracked-cache settings. If
both features are enabled, run one silent status to establish the
provider epoch and persist the complete proof. Do so only when the
caller permits optional locks; never override an explicit
--no-optional-locks request.

Factor the best-effort sidecar reissue used by rebase into wt-status.
After a successful merge commit has installed its final HEAD and index,
use that helper to authenticate the settled state. Sidecar failure still
falls back to ordinary status and never changes the merge result.

Cover receipt publication and adoption after scoped stash, preserve the
generic write-only tempfile contract, and require worktree add to honor
post-checkout index writes, linked-worktree configuration, and disabled
optional locks. Also require a clean non-fast-forward merge to publish a
sidecar that its next read-only status can consume.
@ttaylorr-oai
ttaylorr-oai force-pushed the tb/codex/fsmonitor-hardlink-inodes-unstable branch from 02e1d29 to 8055b6c Compare August 31, 2026 08:59
A command can disable worktree-to-Git filters with empty clean and
process commands and required=false. It does not need to disable smudge,
which converts in the other direction. The status fingerprint only
normalizes the four-setting form, so an otherwise equivalent three-part
override discards scoped FSMonitor history and forces a tracked-file
scan.

Recognize the complete read-side override as well. Continue to
fingerprint partial or mixed-driver overrides, and retain the
normalized-filter bit so that temporarily disabling filters cannot
publish a clean sidecar. The scoped proof must still establish that no
tracked path uses a filter.

Exercise all subsets with both hash algorithms, the three-setting diff
invocation, and active-filter write and priming attempts with either
form.
The reset used by worktree add has no index proof to repair, and its
explicit GIT_WORK_TREE prevents proof authentication. A later optional
status can establish history, but GIT_OPTIONAL_LOCKS=0 suppresses that
priming step. Subsequent read-only status calls cannot persist the
missing proof and keep repeating the tracked-file scan.

When FSMonitor and writable untracked caching are enabled, let the
checkout discover the already registered worktree. Preserve the explicit
repository environment for unsupported contexts, including relative
configuration-file overrides whose meaning would change with cwd.

For a hard reset that creates the index, certify the checkout under the
mandatory index lock before committing it. Attach the current config to
the new index state, query the provider after checkout, and reuse the
writer repair machinery to bind the tracked and untracked proofs. Allow
certification under this owned lock even when optional locks are
disabled; ordinary read-only status still cannot write or repair its
index.

Require a complete untracked scan and a closing provider query. Active
filters, provider errors, and provider resets must leave the checkout
usable without certifying it. Cover those failures, post-checkout edits,
and repeated read-only status calls that reuse the proof without writing
the index or scanning tracked files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant